Sir Christopher Chope says ‘you can’t spend all your time with your family at the same time as being leader of the opposition’